Low-Rank Spatio-Temporal Video Segmentation

Robust Principal Component Analysis (RPCA) has generated a great amount of interest for background/foreground estimation in videos. The central hypothesis in this setting is that a video’s background can be well-represented by a low-rank model. However, in the presence of complex lighting conditions this model is only accurate in localised spatio-temporal regions. Following this observation, we propose to model the background with a piecewise low-rank approximation. To achieve this, we introduce the piecewise low-rank segmentation problem. Starting from a carefully designed cost function which assesses the low-rank coherence of two video regions, the segmentation is obtained with an efficient graph-clustering algorithm. We show that this segmentation, when used to establish a local RPCA per segment, leads to improved quantitative and qualitative results for background/foreground estimation in challenging videos.